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Recent  studies  have  shown  that  criterion  levels  established  In  training 
directly  affect  later  performance  of  subjects  on  experimental  tasks. 
Approximately  20%  of  variance  can  be  explained  by  these  criteria.  The 
purpose  of  this  study  was  to  determine  if  a  similar  relationship  can  be 
found  In  transfer  of  training  situations.  Twenty  male  Air  Force  Academy 
cadets  were  trained  to  one  of  two  multiple  criteria  levels  on  a  difficult 
flight  maneuver  In  a  GAT-1  simulator.  There  was  an  aasy  criterion  set  fnd 
a  more  difficult  criterion  set.  These  two  sets  consisted  of  holding  pre- 
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’scribed  performance  parameters  In  heading,  vertical  velocity,  and  altitude. 
After  achieving  their  assigned  criterion,  all  cadets  In  each  of  the  two 
groups  were  then  tested  on  the  same  task  In  a  GAT-1  simulator,  but  this 
time  the  maneuver  had  to  be  performed  under  turbulent  wind  conditions. 

This  wind  condition  served  as  the  transfer  task.  Half  of  the  cadets  In 
each  group  had  the  same  criterion  In  both  the  training  and  the  transfer 
task.  The  other  cadets  had  different  criteria  In  the  training  and  transfer 
tasks..  Thus  there  were  four  experimental  groups:  easy-easy,  easy-difficult, 
dlffldwt-easy,  difficult-difficult.  One  control  group  had  the  easy  criterion 
while  the  other  control  group  had  the  difficult  criterion.  There  were  five 
cadets  In  each  control  group.  The  dependent  measure  was  the  Transfer  Effective 
ness  Ratio  (TER),  derived  from  trials  of  this  criterion  data.  This  index  Is  an 
estimate  of  the  amount  of  time  saved  In  learning  a  transfer  task  when  perform¬ 
ance  Is  adjusted  to  that  of  a  control  group.  Several  analyses  of  various 
tasks  of  derived  scores  yielded  significant  results,  confirming  that  criterion 
levels  established  in  training  carry  over  to  transfer  of  training  situations. 
Moreover,  the  data  showed  consistency  with  previous  studies  In  accounting  for 
20*  or  more  of  the  variance. 
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INTRODUCTION 


Despite  the  growing  nuaber  of  studies  that  contribute  to  the 
literature  of  flight  simulation,  studies  which  Involve  training  criteria 
as  Independent  variables  are  a  relative  rarity  (Nataupsky,  Schwank, 

Griggs,  McKay  6  Scheldt,  1979;  Nataupsky,  Bernudes  &  Tlraan,  1980).  We 
earlier  demonstrated  that  the  criterion  level  established  for  subjects 
in  training  directly  affected  their  later  performance  level  on  experi¬ 
mental  tasks.  One  result  of  these  training  carry-over  effects  is  to 
bias  the  research  conclusions  reached  by  the  Investigator.  If  carry¬ 
over  effects  are  not  controlled,  they  become  confounded  with  the  effects 
of  the  experimental  variables  and  act  to  lower  the  Internal  validity  of 
the  research  (Campbell  &  Stanley,  1966).  Training  criterion  effects  are 
therefore  Important  factors  for  two  related  reasons.  First,  they  directly 
Influence  the  specific  subject  behaviors  that  will  be  tested  in  the 
research  design.  Second,  the  conclusions  of  researchers  are  Influenced 
by  what  appear  to  be  either  more  or  less  effective  treatments,  but  irtilch 
are,  in  fact,  spurious  effects. 

In  our  experience,  training  carry-over  effects  are  frequently 
overlooked  by  investigators  In  applied  research  and  particularly  In 
flying  training  research.  Perhaps  the  subject  is  given  insufficient 
attention  because  of  the  Inherent  problems  in  working  out  effective 
controls.  For  example,  if  subjects  are  given  a  fixed  number  of  training 
trials,  there  is  the  problem  of  the  more  able  subjects  becoming  more 
highly  trained  than  the  less  able  subjects,  since  they  begin  from  dif¬ 
ferent  ability  levels.  A  fixed  number  of  trials  also  presents  a  problem 
when  subjects  learn  at  different  acquisition  rates.  On  the  other  hand, 
if  subjects  are  trained  to  a  fixed  criterion  level,  there  is  the  problem 
that  the  subjects  with  lower  initial  capability  and  those  with  lower 
acquisition  rates  receive  more  trials  than  the  other  subjects.  Blther 
approach  could  result  in  under  or  over-estimating  the  treatment  effects, 
depending  on  the  ratio  of  high  to  low  ability  subjects.  This  problem  is 
particularly  troublesome  in  flying  training  research,  because  the 
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typical  small  samples  of  subjects  make  it  difficult  to  employ  the 
desired  controls,  such  as  blocking  on  ability.  In  addition,  it  Is 
difficult  to  assess  Initial  ability. 

One  resolution  of  this  problem  Is  to  train  subjects  to  fixed 
criteria,  and  adjust  their  performance  scores  to  that  of  both  controls 
and  their  own  initial  acquisition  rate  to  account  for  differences  in 
achieving  the  criteria.  The  Transfer  Effectiveness  Ratio  (Payne,  1980; 
Roscoe,  1971,  1972)  incorporates  this  concept,  and  this  ratio  can  be 
used  to  determine  the  effectiveness  of  the  training  task  in  preparing 
subjects  to  later  apply  the  training.  In  the  present  case,  this  ratio 
provides  an  index  of  the  relative  strength  of  training  carry-over 
effects  for  the  different  trslning  criterion  levels.  This  ratio  ac¬ 
counts  for  both  initial  capabilities  and  acquisition  rates. 

Conceptually,  the  Transfer  Effectiveness  Ratio  (TER)  is  an  estimate 
of  how  much  in  time  or  trials  to  complete  the  transfer  task  is  saved  by 
the  training  task.  For  example,  if  the  TER  ■  .75,  then  one  trial  in 
training  saves  .75  trials  in  the  transfer  task.  The  higher  the  ratio, 
the  more  effective  the  transfer  of  training.  The  numbers  take  on  added 
significance  when  the  training  is  in  a  simulator  and  the  transfer  task 
is  in  an  aircraft.  Dollar  savings  can  be  projected  from  estimated  costs 
to  operate  both  the  simulator  and  the  aircraft.  The  TER  is  usually 
computed  as  follows: 


S 

x 


Where:  C  »  the  control  group  mean  for  trials  to  criterion  in  the  aircraft. 

X  -  the  experimental  group  mean  for  trials  to  criterion  in  the 
aircraft. 

S  -  the  experimental  group  mean  for  trials  to  training  criterion 
in  the  simulator. 

In  the  present  study: 

C  ■  control  group  mesn  for  trials  to  criterion  in  the  transfer  task. 
Y  -  esch  subject's  trials  to  criterion  in  the  transfer  task. 

S each  subject's  trials  to  criterion  in  trslning. 


2 


w 


Focusing  on  the  problem  of  training  carry-over  effects,  we 
wondered  whether  or  not  training  criteria  Influence  performance  in 
tranefer-of-tralnlng  tasks  in  a  manner  similar  to  the  way  criteria 
effect  experimental  task  performance.  This  question  is  of  Interest 
because  there  are  many  occasions  in  flying  when  the  effect  of  prior 
learning  on  later  application  could  be  hazardous.  The  usual  simulator 
training  is  accomplished  in  "smooth"  air;  transfer  is  then  made  to  an 
aircraft  in  air  with  some  turbulence. 

Following  our  earlier  study,  we  would  expect  to  find  a  dependent 
relationship  between  eventual  outcomes  in  transfer  performance  and 
training  criterion  levels.  In  our  previous  study,  an  average  of  20 
percent  of  the  experimental  variance  could  be  accounted  for  through 
training  criterion  effects.  Estimated  values  of  Omega  squared  were 
used  to  estimate  these  effects.  It  appears  that  the  TER  can  also  be 
used  to  estimate  the  strength  of  relationship  between  training 
criterion  and  task  performance.  Uith  the  TER,  it  is  possible  to 
directly  measure  the  relationship  using  a  transfer  paradigm. 

METHOD 

Subjects 

Thirty  male  senior  cadets  at  the  United  States  Air  Force  Academy 
served  as  volunteer  subjects.  All  subjects  had  completed  a  minimum  of 
15  flying  hours  in  a  T-41C  aircraft  (Cessna  172).  The  median  number 
of  hours  was  25.  Some  cadets  had  had  some  experience  in  other  light 
aircraft.  Each  cadet  was  randomly  assigned  either  to  one  of  four 
experimental  or  to  one  of  two  control  groups. 

Apparatus 

A  Link  Group  General  Precisions  Systems,  Inc.,  General  Aviation 
Trainer  1  (GAT-1)  was  used  to  train  all  pilots  to  selected  criterion 
levels.  The  GAT-1  approximates  the  flight  characteristics  of  the 
T-41C.  This  same  trainer,  using  the  turbulent  air  mode,  was  used  for 
the  transfer  task. 
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Procedure 


Bech  subject  participated  la  two  sessions,  each  one  lasting  not 
more  then  one  hour.  The  first  session  was  a  training  session  and  the 
second  was  the  transfer  task  session.  After  a  review  of  the  cockpit 
Instrumentation,  a  maneuver  called  the  vertical  S-alpha  was  described  to 
the  subjects  using  diagrams  (See  Figs.  1  and  2)  and  a  narrative  ex¬ 
planation.  The  cadets  were  told  to  alternately  climb  and  descend  at  80 
mph  (35.75  m/sec),  500  ft/aln  (2.54  m/sec),  and  to  maintain  a  heading  of 
270  degrees  during  the  entire  maneuver.  After  establishing  a  baseline 
altitude  at  2000  ft  (609.6m)  the  cadets  were  instructed  to  alternately 
climb  and  descend  500  ft  (152.4m)  from  the  baseline  altitude.  All 
subjects  were  told  to  fly  as  closely  as  they  could  to  the  desired 
parameters.  One-half  of  the  subjects  (Easy  Criterion  Group)  met  the 
criterion  when  they  completed  three  successive  trials  where  they  (a)  kept 
their  heading  within  10  degrees  of  the  desired  heading  of  270  degrees,  (b) 
kept  maximum  and  minimum  altitude  within  60  feet  (18.3m),  and  (c)  kept 
the  vertical  velocity  within  300  feet/min  (1.52m/sec).  The  remaining 
one-half  (Difficult  Criterion  Group)  reached  the  criterion  when,  for 
three  successive  trials  they  kept  within  five  degrees  of  the  desired 
heading,  kept  within  forty  (40)  feet  (12.2m)  of  maximum  and  minimum 
altitude,  and  kept  within  200  feet/min  (1.02m/sec)  of  vertical  velocity. 
The  cadets  were  thus  grouped  into  either  the  Easy  or  Difficult  training 
criterion  groups.  All  subjects  from  each  of  these  two  criterion  groups 
were  assigned  to  one  of  two  transfer  task  conditions  (Easy  versus 
Difficult).  Half  of  the  subjects  experienced  a  change  In  criteria  In 
the  transfer  task.  The  transfer  task  incorporated  turbulent  air.  In 
this  manner,  four  experimental  groups  were  formed  with  five  subjects  in 
each  group.  These  four  groups  were  as  follows:  Easy  Criterion-Easy 
Task  (EE);  Easy  Criterion-Difficult  Task(ED);  Difficult  Criterion-Easy 
Task  (DE);  and  Difficult  Criterion-Difficult  Task  (DD).  Two  control 
groups  were  formed,  each  consisting  of  five  additional  subjects.  One 
group  performed  under  the  easy  criterion,  and  the  other  performed  using 
the  difficult  criterion.  Both  control  groups  received  only  the  transfer 
task. 
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Data  Collection 


Data  collection  was  accomplished  unsing  a  hand  scoring  method  to 
record  instrument  readings  on  a  specially  constructed  scoring  sheet  (See 
Fig.  3).  Scoring  was  accomplished  at  selected  portions  of  the  vertical 
S-alpha  maneuver.  All  instructors  participated  in  the  development  of 
the  scoring  sheet  and  had  several  practice  sessions  prior  to  the  data 
collection. 

Scoring 

There  were  five  instructor  pilots.  Each  of  them  recorded  data  from 
one  subject  in  each  experimental  and  control  group.  Thus,  the  scoring 
was  counter-balanced  across  Instructors  and  subjects.  Variables  scored 
Included  altitude,  heading,  vertical  velocity,  and  trials  taken  to  reach 
criterion  during  training  and  during  the  transfer  task. 

RESULTS 

Table  1  presents  the  means  and  standard  deviations  of  training 
trials,  transfer  trials,  and  TER's  which  were  derived  from  the  trials 
data.  In  computing  the  individual  ratios  for  each  subject,  either  the 
Easy  Control  Group  mean  (9.2  trials)  or  the  Difficult  Control  Group  mean 
(14.0  trials)  was  entered  into  the  computation,  as  appropriate.  Each 
individual's  trials  score  was  thus  adjusted  to  that  of  his  appropriate 
control  group. 

A  2x2  analysis  of  variance  (Easy  versus  Difficult  Training  Crl- 
iterion)  X  (Easy  versus  Difficult  Transfer  Task  Criterion)  conducted  on 
the  TER  data  yielded  significant  main  effects  for  both  training  and 
transfer  (p  <.02  and  p  <.04,  respectively).  This  information  is  sum- 
ummarlzed  in  Table  2. 


TABLE  1 


Means  and  Standard  Deviations 

Transfer 

of  Training  Trials, 
Effectiveness  Ratios 

Transfer  Trials,  and 

Groups  (n«5) 

Training 

Transfer 

TER 

EE 

i  e6 

6.0 

.77" 

(4.77) 

(5.61) 

(1.13) 

ED 

5.8 

10.4 

-.29 

(4.21) 

(2.07) 

(.37) 

DE 

9.8 

3.8 

1.66 

(6.61) 

(.84) 

(1.34) 

DD 

6.8 

8.4 

.99 

(1.64) 

(1.57) 

(.59) 

Easy  Control 

— 

9.2 

— 

Difficult  Control 

14.0 

TABLE  2 

Analysis  of  Variance  on  Transfer  Effectiveness  Ratios 


SOURCE 

DF 

MS 

F 

P 

Training 

1 

5.312 

6.437 

.02 

Transfer 

1 

4.232 

5.128 

.04 

Training  x  Transfer 

1 

.094 

.114 

Residual 

16 

.825 

19 

1.202 

DISCUSSION 


Pulg,  Harris,  and  Rlcard  (1978)  outlined  a  number  of  reasons  to 
account  for  recurring  Inconsistencies  In  study  results,  replications, 
and  Interpretations  with  respect  to  the  mo tl on-no  motion  Issue  in 
flight  simulation.  Among  these  reasons  they  listed  Insufficient 
relationship  of  training  objectives  to  criteria,  and  Insufficient 
trsnsfer  of  learning  to  criterion.  No  doubt  many  factors  contribute. 
Including  hardware  dependent  factors.  In  the  present  study,  ws  have 
focused  on  training  factors,  and  particularly  on  the  relationship  of 
training  criterion  to  treatment  effects  and  to  criterion  task  performance. 


The  aein  purpose  of  the  present  study  was  to  investigate  the 
effect  of  training  criterion  levels  on  later  performance  in  a  complex 
flight  task.  Because  an  earlier  study  had  demonstrated  that  training 
criterion  levels  affected  later  perfonumce  levels,  it  seemed  reasonable 
to  expect  similar  carry-over  effects  to  occur  in  transfer  of  training 
situations.  Training  criterion  effecta  often  get  confounded  with  treat¬ 
ments  causing  spurious  treatment  effects  and  wrongful  conclusions  about 
treatments.  It  seemed  possible  that  spurious  effects  might  also  surface 
in  later  transfer  perfonasnce,  and  we  were  Interested  in  estlMtlng 
the  extent  of  the  Influence.  The  TER  was  selected  as  the  index  to 
estlMte  the  effects  since  the  TER  expresses  the  Influence  of  the  train¬ 
ing  situation  in  terms  of  savings  of  time  or  trials  in  the  transfer 
situation.  Additionally,  when  TER's  are  computed  on  an  individual  basis 
subject's  scores  are  adjusted  to  ability  levels  through  comparison  of 
subject  mean  scores  to  control  group  score j. 

Based  on  the  results  of  this  study,  it  seems  reasonable  to  conclude 
that  the  extent  of  influence  of  training  criteria  appears  to  depend  as 
much  or  more  on  the  sequence  of  criterion  difficulty  (easy^dlfflcult  or 
difficult-easy)  than  on  the  level  of  difficulty  Itself.  This  interpre¬ 
tation  is  based  on  a  comparison  of  the  TER  means.  In  comparing  these 
means,  it  should  be  remembered  that  the  higher  the  ratio,  the  more 
effective  the  transfer,  and,  therefore,  the  greater  the  Influence  of 
the  training  criterion  level  on  the  transfer  situation.  However,  this 
rule  is  true  only  in  uncomplicated  cases,  e.g.,  as  in  the  EE  and  DD 
groups,  for  whom  the  training  and  transfer  criteria  were  the  same. 

In  complicated  cases,  e.g.,  as  in  the  DE  and  ED  groups,  the  situation 
is  not  straightforward  because  the  training  and  transfer  criteria  are 
dissimilar. 

It  appears  that  both  sequence  and  difficulty  level  produce  large 
training  criterion  effects.  For  example,  the  best  TER  was  produced  by 
the  DE  group  and  the  worst  was  by  the  ED  group.  The  EE  and  DD  group 
means  represent  only  the  influence  of  difficulty  since  these  two 
groups  were  free  of  sequence  effects.  On  the  other  hand,  the  DE  and 


ED  group  TSR's  reflect  the  combined  influences  of  both  sequence  end 
difficulty. 

The  present  dsts  suggest  thst  both  criterion  levels  end  sequence 
effects  should  be  considered  by  the  resesrch  worker  in  seriously 
accounting  for  experlaentel  vsrlsnce.  Es ties ted  oeegs  squared  vsluee 
for  the  training  and  transfer  aaln  effects  are  .23  and  .19  respectively. 
These  values  sre  consistent  with  findings  in  our  previous  study  that 
showed  training  criterion  effects  accounted  for  an  average  of  20  percent 
of  the  variance.  Consequently,  it  is  likely  that,  to  sone  extent,  both 
difficulty  and  sequence  effects  are  Involved  in  producing  soee  of  the 
variance  referred  to  by  Pulg  et  al  (1978). 
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